Repository navigation
Conversation
On MI300X (gfx942, 304 CU) the flydsl EP v2 backend had no tuned geometry. lookup() fell back to _cu_scaled_default(), which sets combine_block_num to the CU count (304), and op construction asserted `combine block_num 304 exceeds xdb_flag_slots 256`. MI300X HF parts (PCI DID 0x74a9) were not in _DID_TO_MODEL either, so they never reached the MI300X entry at all. Part of ROCm#794 (MI300X only; other untuned parts above 256 CUs still need the clamp). This adds 0x74a9 to _DID_TO_MODEL and a measured MI300X schedule: dispatch 128x8 (128x16 above 1024 tokens), combine 64x8 up to 128 tokens and 32x8 above. Every block_num is <= 128, well under both the CU count and xdb_flag_slots. The MI325X schedule was not aliased, because its combine geometry is 13-19 % slower here at >= 256 tokens. How it was measured: 8x MI300X HF over xGMI, bench_ep.py with BACKENDS=flydsl COMB_MODE=pull MODES=graph, hidden 7168, topk 8, EPR 32, tokens 8..8192, bf16 and fp8 dispatch. Dispatch block {64..304} x warp {4,8,16} and combine block {32..256} x warp {2,4,8,16} were swept per leg. The minimum mean latency was picked per bucket, and the winners and runners-up were re-run twice (spread < 1 % from 128 tokens up). bf16 and fp8 pick the same geometry, so one schedule serves both. EP8 bf16 dispatch+combine pair, us, graph mode, mean of 3 rounds: | tok/rank | this table | MI325X schedule | CU fallback, combine capped 256 | |---|---|---|---| | 128 | 94 | 101 | 108 | | 1024 | 556 | 584 | 788 | | 4096 | 2048 | 2354 | 2975 | | 8192 | 4039 | 4642 | 6020 | Test plan (8x MI300X, gfx942, flydsl default backend, no geometry overrides): ``` torchrun --standalone --nproc_per_node={2,4,8} test_op.py # also TUNED=1, DTYPE=fp8 TUNED=1, HIDDEN=7168 torchrun --standalone --nproc_per_node={2,4,8} test_ep_backend_parity.py # also HIDDEN=7168 TOPK=8 EPR=32 torchrun --standalone --nproc_per_node=8 test_graph_capture.py pytest tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_intranode.py # 17 passed, 1 skipped (fp4, gfx950-only) ``` Authored with Claude.
| # MI350X/MI355X (both 256 CU) are only separable here, not by CU count. | ||
| _DID_TO_MODEL = { | ||
| 0x74A1: "mi300x", # MI300X (gfx942, 304 CU) | ||
| 0x74A9: "mi300x", # MI300X HF (gfx942, 304 CU) |
There was a problem hiding this comment.
what is the diff in the two MI300X version?
There was a problem hiding this comment.
Sorry for the slow reply. They are effectively the same part (gfx942, 304 CU), so they can share the MI300X table. I also mapped the other gfx942 IDs from libdrm's amdgpu.ids that had no model (the VFs and the second MI300X HF ID); the PR description has the list.
detect_model() only knew the base device IDs, so an MI300X VF (0x74b5), the second MI300X HF ID (0x74bd) and an MI325X VF (0x74b9) found no model and, with no gfx942 arch fallback, fell back to the CU-scaled flydsl default that asserts on 304-CU parts. MI308X HF (0x74a8) fell back too and got an untuned geometry. The IDs and names are from libdrm's amdgpu.ids. gfx950 VFs already reach the MI355X table through the arch fallback, so they are left alone. Test: detect_model() with topology() stubbed returns mi300x for 0x74a1/0x74a9/0x74b5/0x74bd, mi325x for 0x74a5/0x74b9, mi308x for 0x74a2/0x74a8. Not run on VF hardware. Authored with Claude.
|
@jeffdaily Thanks for the contribution, and for including the sweep method and results in the table comment. Having a measured MI300X schedule instead of the CU-scaled fallback is useful. I read the diff only (I have no MI300X to run it on), and I have two comments. 1. The default no longer follows the CU count ( The module docstring says The model is picked by PCI DID, and 2. The comment and the first bucket disagree ( The header comment says the gather combine wants block 32 / warp 8 "at every count but 65..128 tokens (64x8)". The first bucket is |
Motivation
On MI300X the flydsl EP v2 backend has no tuned geometry, so a default-constructed op fails (#794).
lookup()falls back to_cu_scaled_default(), which setscombine_block_numto the CU count (304), and construction assertscombine block_num 304 exceeds xdb_flag_slots 256. MI300X HF parts (PCI DID 0x74a9) were also missing from_DID_TO_MODEL, so they never reached the MI300X entry.Part of #794: this fixes MI300X. The issue stays open for the clamp, since any other untuned part with more than 256 CUs still hits the assert.
Technical Details
utils.py: map DID 0x74a9 tomi300x. Only the flydsl v2 lookup changes behaviour; the HIP v2 and internode tables have nomi300xkey, and the v1 path already resolved this part tomi300xby device name.tuning_configs.py: a measured EP8 schedule for MI300X, used for bf16 and fp8 and for hidden 7168, 4096 and 2048 (4096 and 2048 reuse the 7168 schedule, as the MI325X table does). Every block count is 128 or less.The MI325X schedule is not aliased: its combine geometry is 13-19 % slower here from 256 tokens up.
Measured on 8x MI300X HF over xGMI with
bench_ep.py(flydsl, pull combine, graph mode, hidden 7168, topk 8, EPR 32, tokens 8 to 8192, bf16 and fp8 dispatch). Dispatch block {64..304} x warp {4,8,16} and combine block {32..256} x warp {2,4,8,16} were swept per leg, the minimum mean latency was taken per bucket, and the winners and runners-up were re-run twice (spread under 1 % from 128 tokens up). bf16 and fp8 pick the same geometry.EP8 bf16 dispatch+combine, us, graph mode, mean of 3 rounds:
So a clamp alone would leave MI300X about a third slower at large token counts. Below 128 tokens every geometry measured was within run-to-run noise.
Not covered: EP2 and EP4 have no table entry and get the single-shot default (dispatch 128x16, combine 32x8), which passes but is untuned. Only the 0x74a9 part was measured; 0x74a1 shares the table. The runs used flydsl 0.4.0 (an unpinned install today), while
docker/Dockerfile.devpinsflydsl<0.3.A second commit maps the other gfx942 device IDs from libdrm's
amdgpu.idsthat had no model: 0x74b5 (MI300X VF) and 0x74bd (MI300X HF) tomi300x, 0x74b9 (MI325X VF) tomi325x, and 0x74a8 (MI308X HF) tomi308x. Without them, MI300X-class parts with these IDs hit the same assert, and MI308X HF gets an untuned default. gfx950 VFs already reach the MI355X table through the arch fallback. Checked withdetect_model()against a stubbedtopology(); not run on VF hardware.Test Plan
8x MI300X (gfx942), flydsl default backend, no geometry overrides. On main every one of these fails at op construction.
All passed. black and ruff are clean on the changed lines.
This PR was authored with Claude (AI assistant) and reviewed by me.